Speech-assisted facial expression analysis and synthesis for virtual conferencing systems

Fast, reliable, and marker-free facial expression analysis still remains to be a difficult task in computer vision research. In this paper, the concept of speech-assisted facial expression analysis and synthesis is proposed, which shows that the speech-driven facial animation technique not only can be used for expression synthesis, it also provides useful information for expression analysis. From the input speech, the mouth shape can be estimated from the audio-visual model. Thus, the large search space of mouth appearance can be reduced for mouth tracking. Similarly, the modeling technique can be extended from modeling speech and mouth shape to facial movements and detail facial texture changes. In this way, a virtual conferencing system with video realistic avatars can be realized to meet realtime requirement.